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Abstract 

Mean field games is a recent area of study introduced by Lions and Lasry in a series of 
seminal papers in 2006. Mean field games model situations of competition between large 
number of rational agents that play non-cooperative dynamic games under certain symmetry 
assumptions. A key step is to develop a mean field model, in a similar way to what is done in 
statistical physics in order to construct a mathematically tractable model. A main question 
that arises in the study of such mean field problems is the rigorous justification of the mean 
field models by a limiting procedure. 

In this paper we consider the mean field limit of two-state Markov decision problem 
as the number of players TV — > oo. First we establish the existence and uniqueness of a 
symmetric partial information Markov perfect equilibrium. Then we derive a mean field 
model and characterize its main properties. This mean field limit is a system of coupled 
ordinary differential equations with initial-terminal data. Our main result is the convergence 
as TV — > oo of the TV player game to the mean field model and an estimate of the rate of 
convergence. 

D.G. was partially supported by CAMGSD/IST through FCT Program POCTI - FEDER and 
by grants PTDC/MAT/114397/2009, UTAustin/MAT/0057/2008, PTDC/EEA-ACR/67020/2006, 
PTDC/MAT/69635/2006, and PTDC/MAT/72840/2006, and by the bilateral agreement Brazil- 
Portugal (CAPES-FCT) 248/09 

R.R.S was partially supported by the bilateral agreement Brazil-Portugal (CAPES-FCT) 
248/09, and CAPES, PROCAD, Projeto Universal CNPq 471473/2007-3. 

J.M was partially supported by the bilateral agreement Brazil-Portugal (CAPES-FCT) 248/09. 

1 Introduction 

Mean field games is a recent area of research started by Pierre Louis Lions and Jean Michel 
Lasry | LL06a , LL06b , LL07a , LLOTb] which attempts to understand the limiting behavior of 
systems involving very large numbers of rational agents which play dynamic games under partial 
information and symmetry assumptions. Inspired by ideas in statistical physics, Lions and Lasry 
introduced a class of models in which the individual player contribution is encoded in a mean 
field that contains only statistical properties about the ensemble. A key question is how to derive 
such effective or mean field equations that drive the system as well as to show convergence as the 
number of agents increases to infinity. The literature on mean field games and its applications is 
growing fast, for a recent survey see [LLGlOb] and reference therein. Applications of mean field 
games arise in the study of growth theory in economics [LLGlOa] or environmental policy [ALT], 
for instance, and it is likely that in the future they will play an important role in economics and 
population models. There is also a growing interest in numerical methods for these problems 
The authors [GMS10 have also considered the discrete time, finite state problem. 
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In this paper we consider the mean field limit of games between a large number of players 
that are allowed to switch between two states. We are particularly interested in understanding 
the limit as the number of players increases to infinity. We should stress the the fact that we are 
considering only two states plays no special role and we could easily generalize our results to any 
finite number of states. 

In his PhD thesis, GueOS], O. Gueant considered a problem with two states, modeling the 
labor market. In this work he considered a continuum of individuals and a labor market consisting 
of 2 sectors. Each individual has to decide on which sector he or she is going to work. This model 
consists in a coupled systems of ordinary differential equations of the type that will be derived 
in section ||. Another possible application of our models concerns the adoption or change of a 
technology or services. For instance, a single agent faced with different social networks will have 
a incentive to move to the network with more potential contacts, however other effects play a role 
in this player decision, such as the level of services, trouble of changing network, loss of contacts 
and so on. Another similar example concerns switching between cell phone companies. 

We start in Section || to model the N + 1 player problem as a Markov decision process. 
We assume that N of the players have a fixed Markov switching strategy (3 and then look at a 
reference player which looks to minimize a certain performance criterion by choosing a suitable 
switching strategy a((3). This is a well know Markov decision problem. The key novelty in 
this section consists in showing the existence of a Nash equilibrium such that a(f3) = j3 and its 
characterization through a non-linear ordinary differential equation. In fact, this is a continuous 
time, partial information, symmetric version of the Markov perfect equilibrium notion that has 
been studied (mostly in discrete time or stationary setting) in [PS09, Liv02, MT01, 5tr95| ], and 
references therein. In [PM01, SleOl] symmetric Markov perfect equilibrium are also considered, 
and in the last paper the case with an infinite number of players is studied. In [Kap95] the passage 
from discrete time to continuous time is considered for N players in a war of attrition problem. 

In Section [| we derive a mean field model for the optimal switching policy of a reference player 
given the fraction 9(t) of players in one of the states. This model turns out to be a coupled 
system of ordinary differential equations, where one equation governs the evolution of 9, and 
is subjected to initial conditions, whereas the other equation models the evolution of the value 
function and has terminal data. We call this problem the initial-terminal value problem. Initial 
terminal value problems are in fact a general feature in many mean field game problems, see for 
instance [LL06a, LL06b, LL07a . Of course, existence and uniqueness of solutions is not immediate 
from the general ODE theory but, adapting the methods of Lions and Lasry we were successful 
in establishing both. 

Our main result, theorem ^, is discussed in Section ^| where we prove the convergence as the 
number of players N — > oo to a mean field model. 



2 The N+l player game 



In this section we consider symmetric games between N + 1 players under a symmetric partial 
information pattern. We start by discussing the framework of this problem, namely controlled 



Mar kov Dynamics, §2.1, admissible controls §2.3, and the individual player problem §2.4. Then in 
§2.5 we discuss the main assumptions on running and terminal cost that allow us to use Hamilton- 
Jacobi ODE methods, in £2.6 to solve the N+l player problem. Maximum principle type estimates 
are considered in §2.7 which are then applied to establishing the existence of Nash equilibrium 
solutions, 



. This section ends with an example §2.9 



2.1 Controlled Markov Dynamics 

We consider a dynamic game between N+l players that are allowed to switch between two states 
denoted by and 1. We suppose that all players are identical and so the game is symmetric with 
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respect to permutation of the players. To describe the game we will use a reference player, which 
could be chosen as any one of the players. 

If we fix any player as the reference player, we will suppose that he knows his own state at 
time t, given by i(t), and also knows the number n(t) of remaining players that are in state 0. i(t) 
and n(t) are stochastic processes that we will describe in the following. No further information 
is available to the reference player. Because the game is symmetric, the identity of the reference 
player is not important, and all other players have access to the same kind of information, i.e., its 
own state and the fraction of other players in state 0. 

We suppose the process (n(t),i(t)) is a continuous time Markov process: the reference player 
follows a controlled Markov process i(t) with transition rates from state i to the other state 1 — i 
given by (3 = /3(z, n, t). More precisely we have 

w(i(t + h) = l- i\\n(t) = n, i(t) = ij = /3(i, n, t).h + o(h) , 

where lim ^j^- = when h — > 0. Because of the symmetry of the game, all other players follow 
their own Markov process controlled by the same transition rate function j3 : {0,1} x {0, N} x 
[0,+oo) — > [0,+oc). Note that the rate function /3 is a deterministic time-dependent function, 
which makes (n(t),i(t)) a non-time homogeneous Markov process. We will suppose that /3 is 
bounded and continuous as a function of time. We will refer to any Markov control with rate 
function which is bounded and continuous on time, as an admissible control. 
The transition rates of the process n(t) are given by 

j+(i,n,t) = (N-n)p(l,n+l-i,t), (1) 
lp(i,n,t) = n(3(0,n- i,t) , 

where 7^ stands for the transition rate from n to n + 1, and 7^ is the transition rate from n to 
n — 1. Note that n + 1 — i is the total number of players in state 0, as seen by a player (distinct 
from the reference player) in state 1 whereas n — i is the number of players in state as seen by 
a player (distinct from the reference player) in state 0. 
More precisely, we have 

P[n(t + h) = n+l\\n(t) =n,i(t) = ij = j£(i,n,t).h + o(h) , 
p(n(t + h) = n- l\\n(t) = n,i(t) = i) = j^(i,n,t).h + o(h) , 
where lim — when h — > 0. 

h 

We assume further that the state transitions of the different players are independent, condi- 
tioned on i and n. Note that no information is available to any player concerning the state of any 
other individual player. All each player knows is its position and the number of other players in 
state 0, which mean, the fraction of other players in each one of the states and 1. 



2.2 A control problem 



Let now T > 0, and let c : {0, 1} x [0, 1] x R+ -> R and ip : {0, 1} x [0, 1] — > R be two (non-negative) 
functions. We will discuss the precise hypothesis on c and tp in section |2l| We suppose c («, 77, /?) 
represents a running cost incurred by the reference player when he is in state i, n of the remaining 
N players are in state and this player has a transition rate (3 from i to 1 — i. We also suppose 



N . 



represents a terminal cost incurred by the reference player at the terminal time T, if he 



ends up at time T in state i and at that time n of the other players are in state 0. 

If A t (i,n) denotes the event i(t) = i and n(t) = n, the expected total cost of the reference 
player, giving the control /3 and conditioned on the event A t (i,n), will be 



V P (i,n,t) 



A t (i,n) 
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We could be interested in finding an admissible control f3 that minimizes, for each (i, n, t), the 
function V defined above. This however would require a cooperative behavior between players and 
it would be an usual stochastic optimal control problem. Instead, we are interested in finding an 
admissible control j3 that is a symmetric Nash equilibria for the game which we will soon describe. 

2.3 The Dynkin formula 

Given two admissible controls j3 and a, we can define a non-time homogeneous Markov process 
(n(t), i(t)) where the transition rates for n are given by (|l]) and the transition rate for i is given 
by a as 

¥[i(t + h) = 1 - i\\n(t) = n, i(t) = ij = a{i, n, t).h + o(h) , 

where lim 2^1 = when h — > 0. The idea here is that, while other players use the control (3, the 
reference player can choose another control a. 

Furthermore, we have that, for any function ip : {0, 1} x {0, 1, 2, N} x [0, +oo) — >• M, smooth 
in the last variable, and any s > t, 



E Mi,n) [<p{i(s),n(s),s) - tp(i,n,t)] = E^ n) 



— (i, n, r) + A^ ,a ip(i, n, r)dr 
at 



(2) 



where A t (i,n) still denotes the event i(t) — i and n(t) — n, and 

A^' a ip(i, n, r) — a(i, n, r){(p — <fi)(i, n, r)+ (3) 
+ jj(i,n,r)((p(i,n+ l,r) - cp(i,n,r)) +jp(i,n,r)(<p(i,n- l,r) - cp(i,n,r)) , 

where 7^" and 7^ are defined by (Q), and <p(i, n, t) = ip(l — i, n, t). 

We call A a 'P the generator of the process and (||) the Dynkin's formula in analogy to the 
Dynkin's formula in stochastic calculus. 



2.4 Individual player point of view - introducing the game 

Now we suppose the reference player decides unilaterally to use a different control, trying to 
improve its value function. 

We will suppose the other players continue to follow the Markov Chain with transition rate 
/3(i,n,t), bounded and continuous on time. Therefore n{t), the number of such players that are 
in state 0, is a process to which correspond transition rates 7^" and 7J as in (Q). 

The reference player looks for an admissible control a, possibly different from j3, that minimizes 

u(i, n, t, {3, a) = E^ c 

That is, reference player looks for the control a which is a solution to the minimization problem 

u(i, n, t; /?) = inf u(i, n, t, /?, a), 

a 

where the minimization is performed over the set of all admissible controls a. We will call the 
function n, t; j3) above the value function for the reference player associated to the strategy f3 
of the remaining N players. The control a that attains the minimum above can be called the best 
response of any player to a control (3. 
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2.5 Assumptions on running and terminal cost 

We discuss now the hypothesis used in this paper concerning the running and terminal costs. We 
suppose that both the running cost c = c(i, 9, a) : {0, 1} x [0, 1] x — > M and the terminal cost 
tp = 9) : {0, 1} x [0, 1] — > M. are non-negative functions, as mentioned in the previous section, 
and also that they are Lipschitz continuous in 9. Of course, our results would still be valid without 
any change if c and "0 are simply bounded below, instead of being non-negative. 

We assume that c(z, 9, a) is uniformly convex on a > and superlinear. We assume further 
that c is differentiable, and c'(9, a) is Lipschitz in the variable 9. 

For p S R we define 

h(p, 9, i) = min [c(i, 9, a) + ap] . 
Note that h is an increasing concave function of p, Lipschitz in 9, and, hence, bounded below by 

min h(0,9,i). 

0£[O,l],i£{O,l} 

Because of the uniform convexity the minimum is achieved at a single point, and the function 

a*(p, 9, i) — argmin Q > [c(i, 9, a) + ap] . 
is well defined. Furthermore we have 

Proposition 1. The function a* is locally Lipschitz inp, uniformly in 9 G [0, 1]. Furthermore it 
is uniformly Lipschitz in 9. 

Proof. We will use the following inequalities, which are consequence of the uniform convexity of 
c: for all 6,a',a,p andp', we have 

c(9,a')+a'p' > c(9,a)+ap' + (c'(9,a) +p'){a - a) + j\a' - of, (4) 

and because a*(p, 9) is a minimizer, 

(c'(6,a*(p,0))+p)(a'-a*(p))>0. (5) 

We will first prove that a* is uniformly Lipschitz in p : for that, we suppose that 9 is fixed. 
By the definition of a* and equation (^) we have 

c(a*(p)) + a*(p)p' > c(a*(p')) + a*(p')p' > 

> c(a*(p)) + a*(p)p> + (c VGO) +p')(a*(p') - a*(p)) + 7 |«V) - «*(p)| 2 , 

hence 

> (cVGO) +p)(a*(p') - a*(p)) + (p'- P )(a*(p') - a*{p)) + j\a*(p') - a*(p)\ 2 . 
Now using equation (^|) we obtain 

> (p r -p)(a*(p') - a*(p)) + 7 |aV) " ^{ P )\ 2 - 

Therefore 

\p'-p\ \a*(p>) - a*(p)\ > 7 |a*(p') - a*(p)\ 2 , 

which implies 

|a*(p')-a*(p)l< V-Pl- 
7 

This shows that a* is uniformly Lipschitz in p. 
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Now we prove that a* is Lipschitz in 9: for that, we suppose that p is fixed. Again by the 
definition of a* and by equation (^) we have 

c(6',a*{9))+a*(d)p > c(9 ' , a* (6 ')) + a* (6 ')p 

> c(9',a*(9))+a*(9) P + c'(9',a*(9))(a*(9')~a*(9))+ 1 \a*(9')~a*(9)\ 2 , 

and then 

> c'(9\a*(9))(a*(9') - a*{9)) + j\a*(9') - a*(9)\ 2 . 
Using equation (||) we get 

> [c'{9',a*{9)) -c'(9,a*(9))](a*(9') -a* (9)) +j\a*(9') ~a*(9)\ 2 . 

As c'(9,a) is Lipschitz in the variable 9 we have 

> -K\6' - 9\ \a*{9) - a*{9')\ + j\a*(9') - a*(9)\ 2 . 

Therefore 

\a*(9)-a*(9')\ <—\9-9'\, 

7 

which implies that a* is Lipschitz in 9. □ 

In secti on |3.3| we will present and discuss monotonicity assumptions on ip and h, namely 
conditions (|17[) and (|l9|), which will be necessary to prove uniqueness of solutions of the mean 
field model that will be presented in section ||. 

2.6 The Hamilton-Jacobi ODE 

Fix a admissible control (3. Consider the system of ODE's indexed by i and n given by 

=7p(i,n,t)((p(i,n + l,t) - tp(i,n,t)) + 7 J (i,n,t)(cp(i,n - l,t) - ip(i,n,t)) 



+ h n, t) - <p(i, n, t), 



where Lpf){i 7 n,t) = ipp(l — i,n 7 t), and 7^ and 7^ are given by ([TJ). Since 7^ (z, 0, t) = and 
7^~(i, N, t) — 0, the evaluation of ip at n + 1 and n — 1 does not cause problems outside the range, 
resp. when n = N or n = 0). By setting ip n {i,t) — ip{i,n,t) we write the previous ODE in 
compact notation: 

- -J^ =lp (Vn+l ~ <Pn) +7j(^n-l ~ ¥>n) + h((p n - tp n , . (6) 

This system of ODE is called the Hamilton-Jacobi (HJ) ODE for player N + 1 associated to 
the strategy f3 of the remaining N players. We start by proving a verification theorem, which is 



completely analogous to the optimal control verification theorem, see [FS06| for instance. 

Theorem 1. Let ipp be a solution to (g) satisfying the terminal condition ipp(i,n,T) — ip (z, -^). 
Then 

u(i,n,t;/3) = ipp(i,n,t) . 

Also, the control 

a(P)(i,n,t) = a* (tpp(i,n,t) - tpp{i,n,t), , (7) 
is admissible and satisfies 

u(i, n, t; /3) = u(i, n, i, /3, a(/3)) . 
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Thus a classical solution to the HJ equation associated to /? is the value function corresponding 
to (3 and determines an optimal admissible control <5(/3), for the reference player. 



Proof. Let a be an admissible control. By (0) we have 



E' 



^£E{i, n ,r) + A^ a ip (i,n,r)dr 



where A^' a is given by (0). Adding 



A t (t,n) 



T ( n i r ) 

c \i(r), -jj-,a(r) ) dr 



+ <pp(i,n,t) , 



to both sides of the previous identity, where a{r) = a{i{r),n{r),r), and using the definition of 

A^' a (pp(i,n,r), we have 



u(i, n, t; j3, a) 



dt 



(i,n,r) +i+(i,n,r)(tpp(i,n + l,r) - <pp(i,n,r)) 



+ lp (hn,r)((pp(i,n- l,r) - tpp(i,n,r)) +c(i, jf,a) + a(r)(ipp - ipp){i,n,r)dr 

The equation above is valid for all admissible controls a. Now we can define 

a(P)(i,n,r) = a* (tpp(i,n,r) - tpp(i,n,r), , 
which is a bounded continuous Markov control and therefore admissible. We have 

u(i, n, t\ (3) <u(i, n, t, (3, a*) — <pp(i, n, t) 



^A t (i.n) 



~jf( i > n > r } + 7/[ (™,»")( < fti(*,™ + l,r) - <pp(i,n,r)) 



+ 7/3 {n,r)(tpp(i,n - l,r) - ipp(i,n,r)) + h (pp(i,n,r) - <pp(i,n,r), 



dr 



Now, we see that the integrand vanishes since ipp is a solution to HJ, and therefore we have 
u(i,n,t;/3) < ipp(i,n,t). 

Now we prove the other inequality: 

u(i, n, t; 0) — inf u(i, n, t, /3, a) = <fp(i, n, t) 

rT 



infE 



P,a 

A t (i,n) 



+ 7/3 (i, n, r)(ipp(i, n-l,r)- tpp(i, n, r)) + c (i, a) + a(r)((pp - tpp)(i, n, r)dr 



dtp/3 
dt 



(i,n,r) +i£(i,n,r)(tpp(i,n + l,r) - <pp(i,n,r)) 



+ 7/3 (i,n,r)(ipp(i,n- l,r) - tpp(i,n,r)) + h (^p p (i,n,r) - ip {i,n,r), 



dr 
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where the last equation holds because the integrand vanishes since ip is a solution to HJ. 

Thus we have proved that u{i, n, t; ft) = tpp (i, n, t). □ 



2.7 Maximum principle 

Here we prove that the solutions to the Hamilton- Jacobi equations are uniformly bounded inde- 
pendently on the control j3. We denote by 

IK*)||oo = max\iin(i,t)\, 

and 

M = max \h(0,9,i)\. 
(t,e)€{o,i}x[o,i] 

Proposition 2. Let u be a solution to (||). For all < t < T we have 

K*)IU < \\u(T)\\ 0o + 2M(T-t). 

Proof. Let u be a solution to (H). Let u = u + p(T — t). Then 

du n +/ - ~x ~n , f~ ~ n \ 

— ^- = p + lp(u n+1 - u n ) + {u n -x -u n ) + h [u n -u n ,—,i) . 

Let (i, n, t) be a minimum point of u on {0, 1} x {0, 1, • • • , N} x [0, T]. We have u n (i, t) < u n -i(i, t) 
and u n {i,t) < u n+ i(i,t). This implies 7j(u„_i — u n ) > and 7^(u„+i — u n ) > 0. We also have 
u n {i,t) < u n (l — i,t) = u n (i,t), which implies (u n — u n )(i,t) > 0. Hence 

du n ,. . ( - n \ ( n \ 

— ~dT \ n ~ Uni N' ) +P V iV' / +P ' 

because h(p, 0, i) is monotone increasing in p. Furthermore, if we take M < p < 2M we get 

,,<)>». 

This shows that the minimum of u is achieved at T hence 

u n (t,i) > -\\u(T)\\ 00 -2M(T-t). 

Similarly, let (i,n,t) be a maximum point of u on {0,1} x {0,1,-- - , N} x [0,T]. We have 
u n (i,t) > u n -i(i,t) and u n (i,t) > u n+ i(i,t), and this implies 7^"(u n -i — u n ) < and 7^(un+i — 
u n ) < 0. We also have u n {i,t) > u n (l —i,t) = u n (i,t), which implies (u n — u n )(i,t) < 0. Hence 

du n . . f- n \ j n \ 

— ~dT ^ h y u n- u n, —,i)+p<h[0,—,i)+p, 

because h(p, 0, i) is monotone increasing in p. Furthermore, if we take — 2M < p < —M we get 

-7M< 1 

This shows that the maximum of u is achieved at T hence 

u n {t,i) < \\u(T)\\ 00 + 2M(T-t). 

□ 
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2.8 Equilibrium solutions 

We now consider the equilibrium situation in which the best response of any player to a control 
P is p itself. 

Definition 1. Let /3 be an admissible control. This control P is a Nash equilibrium if a(p) = p. 

Theorem 2. There exists a Nash equilibrium, i.e, an admissible Markov control /3* ; which satisfies 
ot(P*) — P*- Moreover, the Nash equilibrium is unique. 

Proof. It suffices to observe that, by (0) 



P*(i,n,t) = a* \yp, - 9/3*,jj,i] , 



and hence the Markov control can be obtained by solving the system of nonlinear differential 
equations 

= 7n( u n+l - Un) + 7~(ti n _l - U n ) + h (u n - U n , , (8) 

with terminal condition u(i, n,T) — tp (i, -^-), where 7^ are given by 

(n + 1 — i \ 
u n+1 -i, — ,1J (9) 

%(i,t) = no* (ti n -i ~ Un-i, 

Note that (@) is well posed because u n is bounded and the righthand side is Lipschitz. Hence it 
follows the existence and uniqueness of a Nash equilibrium. □ 

For the record we give here some properties of 7^: 

< CN, 

and 

l7*+i " In I < C + CN\\u n+1 - Unlloo. 

2.9 An example 

Let / : {0, 1} x [0, 1] — » K and g : {0, 1} x [0, 1] K be two continuous function. We take 

c(«,0,a) = /(*,0) + y -ag(i,6). 

This example could model, for instance, the marketshare of cellular companies where there 
are only two competitors and N individual costumers. If the state of the player represents the 
company he uses, we can think of g(i,ff) as a bonus the company i offers customers of company 
1 — i in case they decide to switch. If there are no such bonus, we set 5 = 0. 

Then 

h{p, 6, i) = min [c(i, 9, a) + ap] = f(i, 9) - ^M-P) + ? ^ 

and 

a*(p, 9, i) = argmin Q > [c(i, 9, a) + ap)} = (g(i, 9) - p) + . 
Therefore (N) becomes 



du {(u-u + g) + ) 2 

= J g + {N-n){u-u + g)l n+1 _ l (u n+1 -u n )+n(u-u + g)^ n _ l (u n ^ 1 -u n ). 

_ _ (10) 

By the results of sectio n p/7| we know that any solution to ([10|) is bounded a-priori. Hence, 
if / and g are Lipschitz, (|10|) has a unique solution u. Therefore, there exists a unique Nash 
equilibrium. 
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3 A mean field model 



This section is dedicated to a mean field model which, as we will see in the next section, corresponds 
to the limit as the number of players N + 1 — > oo. We start in § |3 - 1| by discussing the model and 
its derivation under the mean field hypothesis. Then, in § |3.2| we address existence of solutions. 
Uniqueness of solutions (under a monotonicity hypothesis similar to the ones in JLL064 pLL06bQ 
is established in §3.3. Finally, in §3.4, we continue the study of the model problem from §2.9. 



3.1 The control problem in the mean field model and Nash equilibria 

If the number of players is very large, we expect their distribution between the two states to be a 
deterministic function of the time t, as it would happen if we could somehow apply the law of large 
numbers. So, we suppose the fraction of players in state is given by a deterministic function 
6(t). If all players use the same Markovian control f3 = /3(i,t), which now only depends on i and 
t, then 9 is a solution to 

$L = (1-9)0! -Oft, 6(0) = 8, (11) 

where (3i denotes the function t —> /3(i,t), and < 8 < 1 is given and represents the initial 
distribution. We suppose here that are continuous and bounded, for i = and i = 1, and call 
such controls admissible controls. 

We can now consider the optimization problem from a single player point of view. As before, 
we fix an individual player as the reference player and assume he can choose any admissible control 
a, while other players have a probability distribution among states determined by (Ul[). Let 



u(i,t,a) =E" (()=i 



T 

c(i(s), 9(s),a(i(s),s))ds + ip(i(T), 9(T)) 



where i(t) is a controlled Markov chain switching between state and 1 with rate a. We assume 
this player looks for an admissible control a which solves 

u(i, t) — inf u(i, t, a). 

a 

Note that the situation is now simpler than in the N + 1-player game, because 9 is deterministic 
and the only stochastic process is i(t) whose switching rate is controlled by a. We call u(i,t) the 
value function associated to the mean field distribution 9. 
Consider the following HJ equation: 

du . . „ . . 
- — = h(U-u,9,i). (12) 



As in the verification theorem of §2.6, any solution u to the equation above, with the terminal 
condition u(i,T) = ijj(i,d(T)), is the value function associated to 9. Furthermore, the optimal 
control is a*(u — u, 9, i). 

Under the symmetry hypothesis, all players must use the same control when the Nash equilibria 
is attained. In other words, Nash equilibria is the fixed point to the operator described above, i.e., 
the operator that uses the control /3 to calculate 9 as a solution to (|ll|), and after that determines 
the control a* (u — u, 9, i) where u is the solution to the HJ equation ( |l2|) determined by 9, making 
the control o*(u — u, 9, i) the image of (3 under this operator. 

This leads then to the following system of ordinary differential equations 

= h(u-u,9,i) 

(l-6)a*(u(0,t)-u(l,t),d,l)-Oa*(u(l,t)-u(Q,t),e,0), 
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with the boundary data 

WAV) (i4) 

Note that from the ODE point of view this problem is somewhat non-standard as some of the 
variables have initial conditions whereas other variables have prescribed terminal data. We call 
this the initial-terminal value problem. 

3.2 Existence of Nash Equilibria in the MFG 



We now address the existence of solutions to (13) satisfying the initial-terminal conditions (|14|). 



The proof of existence will be based upon a fixed point argument, using the operator £ described 
in the following, which is the analogous of the operator acting on the controls described in the 
last section, but now acting on distributions. 

Proposition 3. There exists a solution to ( |l3| ) satisfying the initial-terminal conditions (|l4j). 
Proof. We need to solve (13) and (^) which can be rewritten as 



l L = (l-Q) ai -Q a(i 0(0) = (15) 

du 

--^=h(u-u,6,i) u(i,T)=i)(i,6(T)) (16) 

where 

a = a*(u — u, 0, i). 

Let T be the set of continuous functions defined on [0, T] and taking values in [0, 1], with the 
C° norm. Consider the function £ : T — >• T that is obtained in the following way: given € T, let 
u 9 be the solution of equation (filf). Let /3 e = a*(u 9 — u e ,6,i) , and then let £(0) be the solution 
to f = (1 - 0)/3 e - 9/3® and 0(0) = 6. 

From standard ODE theory we know £ is a continuous function from J- to J- . Moreover, as f3 
is bounded, £(0) is Lipschitz, with Lipschitz constant A independent of 0. 

Now consider the set C of all Lipschitz continuous function in J- with Lipschitz constant 
bounded by A. This is a set of uniformly bounded and equicontinuous functions. Thus, by Arzela- 
Ascoli, it is a relatively compact set. It is also clear that it is a convex set. Hence, by Brouwer 
fixed point theorem, £ has a fixed point in C. □ 



3.3 Uniqueness of Equilibria 



To establish uniqueness we need to use the monotonicity method of LL06a, LL06b|. 
We will suppose the following monotonicity hypothesis on ip: 

(x - y)[i>(0,x) - ^(O,?/)] + (y- x)[if>(l,x) - ^(l,y)} > , 



(17) 



for any x and y in [0, 1]. This hypothesis holds, for instance, if we suppose that ip is differentiable 
on its second variable, and 

or if we suppose that ip(0,9) is non-decreasing as function of and ^(i;^) is non-increasing as 
function of 0, which could be interpreted as a penalization on crowded states. 
Now, from the concavity of h in p we have, for all p, q, and i 



h(q, 0, i) - h(p, 0, i) - a* (p, 0, i)(q - p) < 0, 



(18) 



11 



because a*(p,8,i) £ dph(p,6,i). We suppose the additional monotonicity property 



(h(q, 0, 0) - h(q, 6,0))+ §(h(p, 9, 0) - h(p, 9, 0)) (19) 
+(1 - 9) (h(-q, 9, 1) - h(-q, 9, 1)) + (1 - 0) (h(-p, 9, 1) - h(-p, 9, 1)) < - 7 |0 - 0| 2 , 
for all p, q £ R, for some 7 > 0. This property will hold, for instance, if 

h(p,6,i) = h (p)+f(i,9), (20) 

with / satisfying 

(9 - 0)(/(O, 9) - /(0, 9)) + (9 - 0)(/(l, 0) - /(l, 9)) <-j\8- 9\ 2 . (21) 

Note that the example of section [2.9| easily fits the previous conditions ( p0| ) and (||l]) provided 
we suppose g is a constant function and the functions 9 /(0, 0) and /(l, 9) satisfy 



(0-0)(/(G,0)-/(O,0)) > 



and 



7 1 



-0)(/(l,0)-/(l,0))<-^|0-0| 2 , 



which could be seen as a consequence of the fact that the running cost is greater when the reference 
player is in the more crowded state (i.e. when 9 = 1 if i = and when 6 = if i = 1). 
Then , using (18) and (|l9|) we obtain 



(22) 



Theorem 3. Under the monotonicity hypothesis (0) and (|l9]) 7 i/ie system fildi ) and ftlb] ) has a 

unique solution {9,u). 



e(h(q, 9, 0) - h(p, 9, 0) - a*(p, 0, 0)(<? - p)) + 0(%, 0, 0) - h(q, 9, 0) - a*(q, §, Q)(p - q) 
+ (1-9) (h(-q, 9, 1) - h(-p, 9, 1) - a*(-p, 9, l)(p - q)) 
+ (1 - 9) (h(-p, 9, 1) - h(-q, 6, 1) - a*(-q, 6, l){q - pj) < - 7 |0 - 0| 2 . 



Proof. To establish uniqueness we will use monotonicity argument from LL06a , LL06b |. 

Suppose (9, u) and (9, u) are solutions of (|l5| ) and (|l6|). At the initial point t = 0we have that 
(9 - 9)(u - u) = and ((1 - 9) - (1 - 0))(w - u) = 0, where u(i) = u(0,i) and u = u(l,t), and 
similarly for u. Then 



(6-9)(u-u) t = (9 - 9)[-h(u - u, 0,0) + h(u - it, 0,0)], 



and 



((1 - 0) - (1 - 0))(it - u) t = ((1 - 0) - (1 - 0))[-% - U, 9, 1) + h(u - u,6, 1)]. 
Furthermore, 

(u - u){9 - 9) t =(u - u)[(l - 9)a*(u - u, 0, 1) - 9a* (u - u, 0, 0) 
- (1 - 9)a* (u — u, 0, 1) + 0a* (u - it, 0, 0)] , 



and 



(u - u)((l - 9) - 1 + 9) t ={u - u)[9a*(u - u), 0, 0) - (1 - 9)a*(u - u, 9, 1) 

- 9a* (u - u, 0, 0) + (1 - 9)a*{u - u, 0, 1)]. 
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Hence, 



d_ 

dt 



[(9 - 9){u -u) + ((1 - 9) - (1 - 9))(u - 5)) = 



=9( - h{u - u, 8, 0) + h(u - u, 9, 0) + [(u -u)-(u- u)]a*(u - u, 8, 0)) 
+ 9(h(u - u, 9, 0) - h(u - u, 9, 0) + [-(u-u) + (u- u)]a* {u ~ u, 9, 0)) 
+ (l-6)(- h(u - u, 9, 1) + h{u - u, 6, 1) + [(it — u) — (u — u)]a*{u - u, 9, 1)) 
+ (1 - 6) (h(u - u, 9, 1) - h{u - u, 9, 1) + [-(u -u) + {u - u)]a*(u - u, 6, 1)) . 
Then, by using (p^), with p — u — u and q = u — u, we obtain 

■((6» - 9)(u - fi) + ((1 - 9) - (1 - 9))(u - 5)) < -j\6 - 9\ 2 . (23) 



d 

dt' 



Integrating the previous equation between and T, and using the terminal conditions, we have 
that 



T 

2, 



(d(T)-fl(T))[^(0,d(T))-V(0,d(T))]+(e(r)-d(T))[V'(l,d(T))-V(l,d(T))] < -7 / |^(.s)-0( S )| 2 d S . 
Hence by the monotonicity condition ( |l7| ) we get 

< -7 / |6»(s) -0(s)| 2 ds, 



which implies that ^(s) = 9(s) for all s £ [0,T]. Therefore, we have the uniqueness for 9. Then, 
once 9 is known to be unique, we obtain by a standard ODE argument that u — u. □ 

3.4 Back to the example 



Just to illustrate, equations (|13|), in the special case of the example of section and supposing 
g is a constant function, becomes 

d9 

- = (l-9)(u-u)+-9(u-u)+, 

and 

_d^ = {j^-u + g^f 

dt Jy ' ; 2 

As we have already seen, provided the condition (|2l]) holds and given the initial-terminal condition 

9(0) = 6, u(i,T) = iP(i,9(T)) 
the system above has a unique solution. 

4 Convergence 

This last section addresses the convergence as the number of players tends to infinity to the mean 
field model derived in the previous section. 



We start this section by discussing some preliminary estimates in §4.1. Then, in §4.2 we 



establish uniform estimates for |w„+i — u n \, which are essential to prove our main result, theorem 



which is discussed in §4.3. This theorem shows that the model derived in the previous section 



can be obtained as an appropriate limit of the model with N + 1 players discussed in section 
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4.1 Preliminary results 

Consider the system of ordinary differential equations 

- z n = a n (t)(z n+1 - z n ) + b n (t)(z n -i - z n ) + fi n {t)(z n - z n ), (24) 

with a n (t),b n (t), n„(t) > 0. Here z n — (z°,z*), a n — (a°,a*), etc. We assume further that 
a at = and bo = 0. 

We write (Q) in compact form as 

- z(t) = M(t)z(t). (25) 

The solution to this equation with terminal data z(T) can be written as 

z[t) = K{t,T)z(T), (26) 

where K(t,T) is the fundamental solution to ( |25| ) with K(T,T) = I. Note that equations (p5| ) 
and (^6|) imply 

j t K(t,T) = -M(t)K(t,T). (27) 

Lemma 1. For t < T we have 

\\z(t)\\oo < \\z(T)\U 
Furthermore, if z(T) < then z(t) < 0. 

Proof. Let z be a solution of (ps]), and hx e > 0. We define z — z + e{t — T). Hence 2 satisfies 

-z„ = -e + a„(i)(S n+ i - z„) + b n (t)(z n -i ~ z n ) + /J, n (t)(z n - z n ). 

Let (i, n, t) be a maximum point of z on {0, 1} x {0, 1, • • • , N} x [0, T\. We have z n (i, t) > z n -i(i, t) 
and z n (i, t) > z n+ i(i, t), also z n (i, t) > z n (l — i, t) — z n (i, t), this implies b n (t)(z n -i — z n ) < and 
a n (t)(z n+ i - z„) < and fi n (t)(z n - z n )(i,t) < 0. Hence 

This shows that the maximum of z is achieved at T . Therefore, for all (j, m, t), 
Zm(j, t)+e(t-T) = z rn (j, t) < z n (i, T) = z„(i, T) 

Letting e — > 0, we get 

z m (j,t) < maxz„(«,T). 

From this equation we have the following conclusions: 

1. if z(T) < 0, we then have z m (j 7 t) < , for all (j,m,t), and so z(t) < 0; 

2. for all (j, m, t), 

Zm(j,t) < \\z(T)\U 

Now we define z = z + e(T — t). Hence z satisfies 

-z n = e + a n (t)(z n+1 - z n ) + b n (t)(z n -i - z n ) + fJL n (t){z n - z n ). 

Let (i,n,t) be a minimum point of z on {0,1} x {0,1,-- - , N} x [0,T]. We have z n (i,t) < 
z n -i(i,t) and z n (i,t) < z n+ i(i,t), also z n (i,t)_< z n (l - i,t) = z n (i,t). This implies 6„(i)(z„_i - 
z n ) > 0, and a n (t)(z n+ i — z n ) > and fi n (t)(z n — z n )(i,t) > 0. Therefore we have 

-$<*■*> 
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This shows that the minimum of z is also achieved at T, hence for all (J, to, t) 

Zm(j,t) + e(T - t) = z m (J,t) > z n (i,T) = z n (i,T). 

Letting e -» 0, we get 

Zm(j,t) > min z„(i,T). 

n,i 

Hence 

Zm(j,t) > -IKTJUoo. 

Therefore we have ||z(t)||oo < IkC^IU- □ 

Note: let z(t) = K(t, s)z(s) be a solution of ( p5| ) with terminal data z(s) = b, then lemma |l| 
implies that ||^(t)||oo < IN( S ) ||ooj and therefore 

||A'(t,s)6[[ 00 <||6||oo,V6. (28) 

From the previous lemma we also conclude 

Lemma 2. If pi < P2, and t < s, then we have 

K{t,s)p l <K{t,s)p 2 . 

Proof. Observe that if p\ — pi < then K(t, s)(p\ — p-z) < 0, by lemma ^|. □ 
We note now that if t < s < T we have K(t, s)K(s, T) = K (t, T), which implies 



ds 

Hence, using equation (|27]) we get 



^-[K(t,s)K(s,T) I =0. 



-K(t, s)M(s)K(s, T) + ( ^-K(t, s) ) K{s, T) = 0, 



ds 



and therefore, by taking T — s we conclude that 



■^K(t,s) =K(t,s)M(s). (29) 
ds 

We now prove the main technical lemma: 
Lemma 3. Suppose z is a solution to 

-z(s)<M(s)z(s) + f(z(s)). (30) 

Then 

z(t)<\\z(T)\\ 00 + [ H/(*(*))|[ood«. 



Proof. Multiplying ( |30| ) by the order preserving operator K(t, s), we have 
-K(t, s)z{s) < K(t, s)M(s)z(s) + K(t, s)f(z(s)) 

using the identity 

^-K(t, s)z(s) = Kit, s)z(s) + K(t, s)M(s)z(s), 
ds 
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which follows from (|29|), we get 

d 
ds 

Thus, integrating between t and T, we have 



(K(t, s)z(s)) + K(t, s)M(s)z(s) < K(t, s)M(s)z(s) + K(t, s)f(z(s)). 



:(/) - K{I.T)-aT)-_ [ K(t,s)f(z(s))ds. 



So, using equation (E8h, 

i-T 



z{t) < \\z(T)\\ 00 + J Wfizis^Woods. 

□ 

4.2 Uniform estimates 

In this section we prove "gradient estimates" for the N + 1 player game, that is, we assume that 
the difference it n +i — u n is of the order -4 at time T and show that it remains so for < t < T, 
as long as T is sufficiently small. 

We start by establishing an auxiliary result: 

Lemma 4. Suppose v — v(s) is a solution to the ODE with terminal condition 



-f s=C v + CNv 2 + % 
c 



(31) 



where N is a natural number, and C > 0. Then, there exists T* > 0, which does not depend on 
N, such that T <T* implies v(s) < 2£ for all < s < T. 

Proof. Note that (|3ll ) implies that v is a monotone decreasing function of s and is equivalent to 

{— = -l 
s(§) < T. 

This implies by direct integration that 

,'2C\ f& dv 



N J ~ Jc Cv + CNv 2 + % ' 
Now 

2C 2C 

> / — ^ ^ ^cfo 



/; 



Cv + CNv 2 + %- Jc 2C 2 + AC 3 + C~" 2C + 4C 2 + l' 

2C> 



ICS 



Therefore if we define T* — 2 c+lc 2 +i > we nave that s(^) < if T < T*. Hence this impli 
v(0) < which yields the desired result when we take into account that v is a decreasing 
function of s. □ 

Proposition 4. Suppose that 

K+l(r)-Un(T)ll°o< ^ (32) 

/or C > 0. Lei u be a solution of (||). T7ien i/iere exists T* > swc/i i/iai, /or < T < T* ,we 
have 

2C 

\\u n +l{t) ~ Mn(f)||oa < -jy") 

/or allO<t<T. 
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Zn ^n- 



Proof. Let 

Note that, as usual, z n — (z n ,z n ). We have 

~Zn = In+lZn+l ~ 7n Z n ~ ln+l Z n + In z n-l + k ^ 



n + 1 



N 



Un+l — Un+1 



(n . 



We can write 



ln+l Z n+l ln z n ln+l Z n + 7n z n-l 

Tn+1 In i \ Tn+1 — "tn / s 

— 2 i^ri+l — ^nj i ^ + Zn) 

, 7n+l 7n / \ . Tn Tn+l / , \ 

H -Z (Zn-l - Z n ) H + Z n J- 



We must now observe that 



as well as 



Furthermore, we have 




< C + CN\\z\U 



< C + CN\\z\\ c 



n + 1 



N 



U n +1 - Un+1 



= 1 1 I ^ ,i,U n+1 ~ Un+lj - h ^—,i,U n +l - U n+ 1 



c 

"t - f-^n\Zn Z n )i 

where ^ n = h p (-^, i, u n - u n ) > 0. 

At this point we are in position to apply lemma [| from the previous section. We obtain 

Zn(t) = (Un+l - U n ){t) < ^(T)^ + J + C||z( S )||^ + ^ ds . 

We can also use the same argument applied to 

Zn t*7l • 

Finally, if we set w = \\u n+ i — u n ||oo we conclude that 



;/•(/;• ,r(T)-\ / Cw(s)+CNw(sf + ^ds. 



Now we define 



//(/) = <c(D- / Cw(s) + CNw(s) 2 + ^ds. 
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We have that 
and also that 



w(t) < v (t), 



(33) 



drj 



(t) = -g(w(t)), 



dt 

where g is the nondecreasing function g(w) = Cw + CNw 2 + jj. Thus 

{%{t)>-g{r,{t)) 
\r,(T) = w(T). 

A standard argument from the basic theory of differential inequalities can now be used to prove 
that rj(t) <v(t) for < t < T if v(t) is the solution of 

/&(*) = -*(«(*)) 
\v(T) = w(T). 

This last result can be combined with lemma|, the hypothesis w(T) < & and the inequality (|33|), 
to prove that w(t) < for all < t < T, which ends the proof of the proposition. □ 

4.3 Convergence 

In this section we prove theorem ||, which implies the convergence of both distribution and value 
function of the N + 1-player game to the mean field game, for small times. 

We start by assuming that at the initial time the N players distinct from the reference player 
distribute themselves between states and 1 according to a Bernoulli distribution with probability 
6 of being in state 0. 

Let 



V N (t)=E 



6(t) 



W N {t) =E[(u(0,i)-u n(t) (0,i)) : 
W N (t) = E \{u{l, t) - u n(t) (l,t)Y 



(34) 



[Q N (t) = W N {t) + W N (t), 
where 9(t) is the solution of (|lT|), < n(t) < N is the number of players (distinct from the 



reference player) which are in state at time t, and u = u(i,t) and u n = u n (i,t) are respectively 
the solution of the H J equation and terminal conditions for the MFG (|3|) and N + 1 player game 

®- 

We have 



V N (0) = Var 



n(0) 



N 



1 



N 



because n(0) is the sum of N iid rv with Bernoulli distribution. 

In this section a — a(i,t) is the optimal control for the MFG, whil e a = a N (i,n,t) is 

the optimal control for the N + 1 player game. We know from sections ^5, 2J3 and [O] that 

= a* (u„ — u n , j], i) and a — a*{u — u, 9, i). 

Lemma 5. There exists C\ > such that 



V N (t) < 



C 1 (V N (s)+Q N ( s ))ds + ^. 
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Proof. Using Dynkin's Formula (^|) with tp(i, n, s) — (^j^- — 0( s )j j we have 



where 

uj n ( s ) = (iV-n)af 



V N (t) - MLJ£l = E I WJV ( 8 ) + , N{s)ds 



o , 



AT 

aj = a I u n+ i-i - Un+i-u 



N 

n + 1 — i 



N 



1 , 



W„ = u N {i,n,t) , 



and 



We have 



= ^(i, », r) = -2 - (?) ((1 - 0)a x - 0a o ) • 



Ww (*) = (l-^)af 



n \ w / 2n + 1 



-26 - 



=2a 



7i N ( 2n — 1 
n / n 



where tat(s) = + -^{olq - af ) . Now 



vn(8)+s n (s) = 2 - 9) [af (l - 2L) - - ((1 - fl)ai - 0a o )] + t*( S ) 



Tiv(s) 



- 2(af + O - *) 2 + 2 - fl) ((ai - af + a - af )0 + (af - ax)) + t*(*). 



Then 



VKr(t) 



)(1- 



N 



= - 2E 



+ E 



((«i - af + ao - af )0 + (af - a x )) ds 



T]y(s)ds 



Now we see that 



|a - af | 



a*(tt — u, 6, 0) — a* I u n 



Tl 2 



< X 

< if 

< if 



iV 



7; 
JV 
n 
N 



+ \u — U n -i\ + \u— u„_ 4 | 



3 



n| ~r — i T ^ 



+ U — U n + \U - Un\ + 



iV 
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where we used that a* is Lipschitz in both variables, and u and u N are bounded, and the uniform 
bounds on |u n +i — u n \ obtained in proposition^ of §4.2. Similarly 



|ai - of | < K 



n 

N 



+ \u- u„\ + \u- u n \ + — 



Thus 





H 












N 



V N (t) <K l / V N (s)ds + 2 
Jo 

<(K 1 +2K)J^ V N (s)ds + 2Ej ^-6^JK 



K 2 

+ \u — u n \ + \u - u n \ + — )ds + —— 



N 



N 



K 2 

\u - u„\ + \u — u n \ + )ds+ — 



<{K 1 + 2K) V N (s)ds + 2K 2V N (s) + (W N (s) + W N (s))ds + 



K 2 + 6T 
N 



K 3 V N (s) + 2KQ N {s)ds 



K 2 + 6T 



n 



N 



<J^ C 1 (V N (s) + Q N {s))ds + ^ 



Lemma 6. There exists C 2 > such that 

i-T 



□ 



Q N {t)<J C 2 (V N (s) + Q N (s))ds+ < ^-. 

Proof. In this proof, u n (s) or simply u n will denote the expected minimum cost of player N + 1 
conditioned on its state being equal to at time s, i.e., u„( s )(0, s). We will also use, here, u(s) or 
simply u to denote u(0, s). 

Using Dynkin formula (||) with ip(i, n, s) — (it„( s ) (0, s) — u(0, s)) , and equations (||) and (|l2|), 
we have 

W N (t) - W N (T) = -E[(u n (t) - u(t)) 2 ]+E[(u n (T) - u(T)) 2 } 



=E / 2{u n — u) — (u n — u)ds 

+ E 7+ [(u n+1 - u) 2 - (u n - u) 2 ] + 7„ [(u n -i - u) 2 - (u„ - u) 2 ] ds 



--E I 2(u n - u) I -7+ (u n+1 - Un) - 7„ {u n -i - U n ) - h\u n - u n , — , J + h(u - u,9,0) ) ds 
+ E / In [K+i - u) 2 - (u n - u) 2 ] + 7„ [(u n -i - u) 2 - (u n - u) 2 ] ds 



7+(w„+l - U n ) 2 + 7„ (U n -i - U n ) 2 - 2 (h (u n - U n , >°) ~ H u ~ u i °)) i u n ~ u)ds, 

where 7^ = 7„ (0, n(s), s). In the last equation we used the fact that 

-2(u n - m)7+(u„+i - U n ) + In [( u n+l ~ u) 2 - (u n - u) 2 ] = 7+ (u n+1 - U n ) , 

and a similar calculation for 7". 

Now, using results from §4.2, proposition^, we have that 7+(u ra +i — u n ) 2 , ~f~{u n -\ — u n ) 2 and 
Wn(T) are bounded by which implies 

W N (t) < ^ + 21 



+ 2E (h (u n - u n , —h{u— u, 9, 0)) (u„ - u)ds 
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Using the fact that h is Lipschitz in both variables, we have 

h (u n - u n , - h(u - u,6,0) < 6 - + \u-u„\ + \u- u„|^ 







N 


V N (s) + 


W N (s) 



Thus 

W N (t)<^+K 

With a similar calculation we have a analogous inequality for Wiq{t), which ends the proof. □ 

Now we can state and prove our main result that establishes the convergence of the N + 1 
player game to the mean field model as N — > oo. 



Theorem 4. If p = TC < 1, where C — max{Ci, C2}, and Qn{1) + V/v(i) is given in ^84 ) th 

Q N (t) + V N (t)<-^—^ Vte[0,T]. 

1 — p N 

Proof. Adding both inequalities given in the two last lemmas, we have 
Q N {t) + V N {t)<C [ (V N {s) + Q N (s))ds + C 



N 



Now suppose p — TC < 1. Defining 



Qn + V N = max Q N (t) + V N (t), 

0<t<T 

we have 

C 

Qn + V n < p{Q N + V N ) + —, 
which proves the theorem. □ 
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